On the Inefficient Use of Entropy for Anomaly Detection

Entropy-based measures have been widely deployed in anomaly detection systems (ADSes) to quantify behavioral patterns [1]. The entropy measure has shown significant promise in detecting diverse set of anomalies present in networks and end-hosts. We argue that the full potential of entropy-based anomaly detection is currently not being exploited because of its inefficient use. In support of this argument, we highlight three important shortcomings of existing entropy-based ADSes. We then propose efficient entropy usage – supported by preliminary evaluations – to mitigate these shortcomings. 1 Entropy Limitations and Countermeasures 1.1 Feature correlation should be retained Current ADSes perform entropy analysis on marginal distributions of features. In general, significant correlation exists across traffic and/or host features which is not being leveraged by these ADSes. As a proof-of-concept example, we propose to detect malicious network sessions by noting that the histogram of keystrokes which are used to initiate network sessions is skewed [see Fig. 1(a)] and perturbation in this metric can easily reveal the presence of an anomaly; network traffic and keystroke data were collected before and after infecting a human-operated computer with the low-rate Rbot-AQJ worm. While analyzing the entropies of the marginal keystroke distribution and/or the marginal session distribution is clearly not useful, Fig. 1(b) shows that quantifying these features using joint (session-keystroke) entropy can easily detect anomalous activity. 1.2 Spatial/temporal correlation should be retained Another limitation of the entropy measure is its inability to take spatial/temporal correlation of benign patterns into account. Such correlations can prove useful in the detection of subtle anomalies. For instance, Fig. 1(c) shows the blockwise (block size = 1KB) entropy of a PDF file which is infected by an embedded executable malware. It is evident that entropy is unable to provide clear perturbations required for detection. On the other hand, entropy rate [Fig. 1(d)], which models and accounts for the spatial/temporal correlation, provides very clear perturbations at the infected file blocks; entropy rate quantifies the average entropy of conditional distributions [2]. 1 13 40 63 9 32 65 2 162 38 0 0.2 0.4 0.6 0.8 Virtual Key Code N o rm a liz e d F re q (a) Histogram of session-keystrokes 200 400 600 80